Learning systems and automatic transitioning between learning systems

ABSTRACT

An online learning system selects content for a learning session and opens a user interface to start the learning session on a plurality of devices. A mode of instruction is selected for the learning session. An activity to perform associated with the content is presented. Performance of the activity is monitored and a performance metric and/or a heterogeneity metric associated with a key performance indicator for the activity performed is generated. Responsive to determining that the performance metric and/or a heterogeneity metric is outside of the target range, the mode of instruction may be switched automatically.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/475,248 filed Mar. 31, 2017 which is incorporated by reference hereinin its entirety.

FIELD

The present application relates generally to computers and computerapplications, and more particularly to learning systems.

BACKGROUND

Computer-based training (CBT) often implies self-paced instructionavailable anywhere and anytime at the convenience of the learner;however, CBT methods can complement and provide many advantages even toconventional methods of classroom instruction at a centralized location,in which the students and instructor must meet physically in the samelocation at the same time. On the one hand, self-paced technology-basedinstruction is one-to-one, so that every student can receive materialtailored to individual needs and proceed at his or her own pace.However, this method can also be highly disadvantageous if the goal ofthe learning activity includes the maximization of utilization of thegroup as a whole (and not of a particular individual). On the otherhand, in instruction-led technology-based instruction methods theinstructor controls the speed of the class and manages the groupproficiency level, but it is often the case in which he or she bypassesslow learners or bore fast ones, mainly due to the lack of real-timefeedback describing the performance of the students. As a result, manystudents may feel overwhelmed whereas some may receive training they donot need.

BRIEF SUMMARY

A system and method of online learning may be presented. The system, oneaspect, may include at least one hardware processor and a storage devicecoupled to the hardware processor. The hardware processor may selectcontent for a learning session and open a user interface to start thelearning session on a plurality of devices. The hardware processor mayselect a mode of instruction for the learning session. The mode ofinstruction may include self-paced learning or mediated learning. Thehardware processor may present via the user interface an activity toperform associated with the content during the learning session. Thehardware processor may retrieve from the storage device a keyperformance indicator associated with the activity, a target rangeassociated with the key performance indicator, and a heterogeneitythreshold associated with the activity. The hardware processor maymonitor performance of the activity, and generate a performance metricassociated with the key performance indicator for the activity performedon each of the plurality of devices based on the monitoring. Responsiveto determining that the performance metric is outside of the targetrange, the hardware processor may automatically switch the mode ofinstruction to a different mode of instruction, and present on the userinterface, the activity to be performed in the different mode ofinstruction.

A method of online learning, in one aspect, may include selectingcontent for a learning session and opening a user interface to start thelearning session on a plurality of devices. The method may also includeselecting a mode of instruction for the learning session, wherein themode of instruction may include self-paced learning or mediatedlearning. The method may further include presenting via the userinterface an activity to perform associated with the content during thelearning session. The method may also include retrieving from a storagedevice a key performance indicator associated with the activity, atarget range associated with the key performance indicator, and aheterogeneity threshold associated with the activity. The method mayalso include monitoring performance of the activity, and generating aperformance metric associated with the key performance indicator for theactivity performed on each of the plurality of devices based on themonitoring. The method may further include, responsive to determiningthat the performance metric is outside of the target range,automatically switching the mode of instruction to a different mode ofinstruction, and presenting on the user interface, the activity to beperformed in the different mode of instruction.

A computer readable storage medium storing a program of instructionsexecutable by a machine to perform one or more methods described hereinalso may be provided.

Further features as well as the structure and operation of variousembodiments are described in detail below with reference to theaccompanying drawings. In the drawings, like reference numbers indicateidentical or functionally similar elements.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow diagram illustrating defining of various parameters foronline learning in one embodiment of the present disclosure.

FIG. 2 is a flow diagram illustrating a method of the present disclosurein one embodiment.

FIGS. 3-5 illustrate examples of learning sessions performed by usersvia their devices in one embodiment of the present disclosure.

FIG. 6 illustrates a schematic of an example computer or processingsystem that may implement a learning system in one embodiment of thepresent disclosure.

DETAILED DESCRIPTION

A method and a system may be provided for a learning system to recommendwhen the instruction modes should be switched between self-pacedlearning and instruction-led technology-based learning to maximize theefficiency (e.g., time spent to learn a certain concept, success rate onevaluation steps) of a group of students during a learning session. Thelearning system of the present disclosure in one embodiment mayorchestrate a learning session, for example, by controlling differentmodes of operation such as self-paced learning (asynchronous learning)and mediated learning (synchronous learning). The system in oneembodiment may monitor online learning sessions, assess in real-time theperformance of learners in a group, and autonomously indicate anexecution mode that optimizes the learning of the learners in the group.

In one embodiment, the method and system are capable of recommending inreal-time a learning method to employ during a point in a learningsession in order to maximize the efficiency of the group's learning. Thelearning system of the present disclosure in one embodiment may startlearning session with all students or learner in a self-paced learning.For example, an individual learning session (e.g., a web site's browserpage or a learning application) may be opened for each learner and eachlearner allowed to independently execute through the learning session atthe learner's own pace. The learning system may capture key performanceindicators (KPIs) of individuals. Examples of KPIs may include but arenot limited to, execution time in each step, skill level, scores onquizzes. Responsive to determining that one or more of those performancemetrics (KPIs) one or more individuals taking part of the learningsession falls under the expected thresholds, a notification isgenerated. The notification may include a signal to an instructorindicating potential difficulty a student (learner) or a subgroup ofstudents has in the self-paced method. This may occur, for example, if astudent is spending relatively more time than expected in a topic ormaking many mistakes on evaluation steps. The notification may include arecommendation to switch to technology-mediated instruction mode. In oneembodiment, the notification may include a signal to a system of thepresent disclosure to automatically switch the learning mode of thestudents to the technology-mediated instruction mode. Thetechnology-mediate instruction mode, for example, is instruction-led. Inthe instructor-led technology-mediated instruction, once the efficiencyof the student or the subgroup that was below the threshold catches up,i.e., exceeds the threshold our boundary of the target metrics (e.g.,success rate of students exceeds the minimum threshold level), thelearning system of the present disclosure generates a notificationsignaling that the learning session can be switched back to that of theself-paced mode. In one embodiment, the notification may include asignal to a system of the present disclosure to automatically switch thelearning mode back to the self-paced mode.

Repository of educational material stores a collection of learningobjects for one or more learning activities. Learning objects mayinclude content and assessment items.

A database of metadata store metadata associated to each individuallearning object. Examples of metadata may include, but are not limitedto: data describing the difficulty level, expected completion time,prerequisites, acquired competencies or skill points, target learningstyle. Metadata may be added by the author of the learning objects,automatically derived from historical data. For example, averagecompletion time of the learning content or material may be computedbased on past completion time of students who have successfully learnedbased on the learning content.

A database of students' profile stores students' profile (orclusterization) containing information about preferred learning styles,current knowledge levels or skill points, strengths, weaknesses of arespective learner.

A learning plan in one embodiment contains a collection of concepts andexercises (e.g., in the form of digital learning objects) to be taughtin a learning session. The instructor can create such plan eithermanually or with support of an automatic method and system.

Each student or learner has a device (e.g., smart phone, tablet or PC)to interact with the learning objects of a given learning session.Before learning session starts, the system receives target learningplan, students' profiles (e.g., current knowledge level), andinformation describing the desired threshold levels for performanceindicators (e.g., execution time in each step, performance inevaluations).

FIG. 1 is a flow diagram illustrating defining of various parameters foronline learning in one embodiment of the present disclosure. One or morehardware processors may perform the method. At 102, content to bepresented in a learning session may be defined and/or generated. Thecontent, for example, may be received from a storage device storinglearning session content. For instance, the content may be definedbefore the learning session by an instructor or a teaching institution.A content is any topic that the students should learn in a specificcourse, for example, in a Geometry course, each learning session mayfocus on topics such as surfaces, points, curves, distances.

At 104, activities associated with the content may be defined orgenerated. Examples of activities may include, but are not limited to,definition of terms, text material, multimedia content, quizzes,questionnaires, and tests. The activities provide learning environmentfor learning the content during the online learning session. Briefly,the content refers to the material received by the student; the activityrefers to the task for the student to practice how well he understoodthe material, for example, which is monitored by the systemautomatically to access the performance of the students.

At 106, key performance indicators (KPI) and target KPI rangesassociated respectively with those key performance indicators may bedefined. KPIs are indicators or measures that allow identifying thoselearners who are excelling or not doing well on the content or topic,during an online learning session. Examples of KPIs may include, but arenot limited to, error rate, time to perform a given task, and accuracyrate. Target KPI ranges can be defined that provide criteria orthreshold of determining whether a learner (student) is doing well ornot. In one embodiment of the present disclosure, a KPI range can bedefined based on the historical data, considering the higher range valueas the mean plus standard deviation of the said KPI and the lower valueas the mean minus the standard deviation of the said KPI.

At 108, a heterogeneity threshold applicable to a group of learners(e.g., a class) may be defined, for example, based on the KPIs. Forexample, the heterogeneity threshold may be pre-set by an instructor orteaching institution and defines the minimum KPI value acceptable for astudent in a given content (different contents may have differentthresholds). In another embodiment, the threshold may be determinedbased on historical data, for example, defined minimum test grades orKPI values associated with a student at the time the students ask theinstructor for clarification on a given subject. The defined orgenerated information at 102, 104, 106 and 108 are stored in a databaseor the like, in a storage device.

FIG. 2 is a flow diagram illustrating a method of the present disclosurein one embodiment. One or more hardware processors, e.g., executing anonline learning system (or module), may perform the method. At 202, alearning session is started. For instance, content may be selected forthe learning session and a user interface or a browser window is openedon a plurality of devices. Users on the plurality of devices make up aclass or a group for the learning session.

At 204, an instruction method may be selected. For instance, a mode ofinstruction for the learning session is selected, for example, from aself-paced learning mode and a mediated learning mode. In self-pacedlearning mode, a user on a device (e.g., of the plurality of devices)runs through the content for learning during the learning session onuser's own pace, asynchronously from other devices (e.g., of theplurality of devices). In mediated learning mode, the pace of learningis synchronous among the plurality of devices. For example, theplurality of devices run through the content of the learning session atthe same pace as the learning in that mode is synchronized. In oneembodiment, the instruction mode selected automatically by a hardwareprocessor running the learning session, for example, based on historicaldata associated with the learning capabilities of the group that ismaking up the class, for example, student information that may includethe knowledge level and speed or rate of learning associated with thestudents in the group, learning plan. The learning session may bestarted in the mediated mode (e.g., assuming that, for the currentcontent, the KPI level of the classroom is zero). In another embodiment,the system may use historical data (from the current classroom or fromother classrooms that had previously received the same content) to startthe learning session on a different mode.

In one aspect, the online learning session (e.g., executing on ahardware processor) may send a notification to a user such as aninstructor that is leading the online session, for example, via a userinterface of the learning session that is running on that user's device,the selected mode, and the user may be given an option to accept oroverride (e.g., select another instruction mode) to start the learningsession.

At 206, one or more activities associated with the learning content arestarted on the online learning session. For example, one or moreactivities to perform are presented via the user interface running onthe plurality of devices.

At 208, students perform the one or more activities presented in theonline session, and the performing of, e.g., interacting with, those oneor more activities are monitored

At 210, one or more KPIs associated with the one or more activities, atarget range associated with the key performance indicator, andheterogeneity threshold associated with the one or more activities areretrieved, for example, from a storage device 228 that stores theinformation. In one embodiment, this module 228 stores: the algorithmfor computing the performance of the students during the activity basedon the collected KPIs (212), the heterogeneity threshold that ispre-defined, for example, before the learning session (214), thealgorithm for computing the topics in which the students are notperforming well (218), and the mapping of the most adequate instructionmode according to the current performance (220). The system provides asuggestion based on this data, and in one embodiment, the instructor maybe provided with an option to accept or not accept the suggestion. Thesystem stores in the same module 228 the information whether the useraccepted the suggestion (222), for example, which the system may use torefine the algorithms.

At 212, based on the monitoring of the performing of the activities, aperformance metric respect to the one or more KPIs is determined, forexample, for the activity performed during the learning session. Aperformance metric is determined for an activity performed during alearning session on a device, for example, associated with a student orlearner. Each of the plurality of devices (running a learning sessionactivity) generally might not have the same performance metric, as onestudent's pace of learning may be different from another student's paceof learning. For instance, the monitoring may include determining theamount of time a user spends on an activity, for instance, computes thetime the activity is presented on the user interface until a nextactivity is presented on the user interface, or for example, until auser input is received on the user interface. Since the tests areexecuted in a computer device, the system may check how long eachstudent took to complete the test. For example, if each test is loadedon a different screen, the system stores the timestamp of when thestudent first loaded the screen, and the timestamp of when the studentleft or closed the screen. Also, the system stores the ground truth(previously defined) for all tests, and can check whether the studentshad inputted the same answer from the ground truth. In this way, theonline learning system of the present disclosure may identify whichstudent is doing well or not well on which activities or topics.

At 214, the online learning system determines whether the performancemetrics computed for all students in the group exceed the heterogeneitythreshold. For example, based on the KPI, the system computes aperformance metric (which is a numerical value) for each student. Thesystem analyzes whether these values are similar for all students, bycomputing, for example, the standard deviation, which produces anothernumerical value—this second value is then compared with theheterogeneity threshold (previously stored) in order to estimate whetherthe students are performing similarly, for example, within the standarddeviation.

If the heterogeneity threshold is not exceeded, at 216, the onlinelearning system determines whether one of the student's performancemetric is below the KPI range. If it is determined that the students areperforming similarly (e.g., as determined at 214), there may be twopossible scenarios: (1) all students are performing well enough or (2)all students are not performing well enough on the topic.

If it is determined that the heterogeneity threshold is not exceeded at214 and that a student's performance metric is not below the KPI rangeat 216, the logic of the method proceeds to 208, and the online learningsystem continues to monitor performance of one or more activities in thelearning session.

If at 214, the online learning system determines that the performancemetrics computed for all students in the group does not exceed theheterogeneity threshold, or at 216, the online learning systemdetermines that a student's performance metric is not below the KPIrange, the logic of the method proceeds to 218.

At 218, the online learning system consolidates information associatedwith one or more topics in which the student or students are facingdifficulties. For example, a learning session may be composed of a groupof topics. The learning system may evaluate the performance of thestudents on each of the topics, and for example, determine or inferwhich students are not meeting a performance threshold.

At 220, the online learning system determines an alternative instructionmode that would help in allowing the student or students facingdifficulties to be able to more easily comprehend the topic ofdifficulty. For example, if all students are performingunsatisfactorily, the system may suggest and the instructor may use themediated mode to provide more detailed explanations on the problematictopics determined by the online learning system.

Learning modes that are suggested may include self-paced, mediated,group activity, and pair activity. For instance, if the current mode ofinstructions is mediated learning, then the alternative instruction modemay be one of self-paced learning, group activity learning, and pairactivity learning. As another example, if the current mode ofinstructions is self-paced learning, the alternative instruction modemay be one of mediated learning, group activity learning, and pairactivity learning.

At 222, the online learning system notifies a user (e.g., user's devicethat a user leading the online learning session is using), for example,an instructor, that the instruction mode should be switched. In oneaspect, the system provides an option to the instructor to not acceptthe suggestion to switch the mode, for example, via a user interface onthe user's device. For example, at 224, responsive to the instructoroverriding the suggestion or not accepting the suggestion, the onlinelearning system registers (or stores) the information that the suggestwas not accepted and returns to the monitoring of activity performanceat 208.

At 226, responsive to the instructor accepting the suggestion, theonline learning system switches the learning mode to the suggested mode(alternative instruction mode). In one embodiment, the online learningsystem may automatically or autonomously switch the instruction mode,for example, without an input from the instructor. The logic of themethod proceeds to 206, where an activity is presented based on theswitched instruction mode.

In one embodiment, independently from the chosen strategy, the onlinelearning system may monitor and collect information about targetparameters in real-time (e.g., execution time per step, performance onevaluation steps). On self-paced mode, whenever the performance metricof a particular individual or a subgroup degrades and falls under theexpected threshold, an instructor or the like may be notified aboutpotential difficulties the students are facing (e.g., spendingconsiderably more time than expected in a topic or making many mistakeson evaluation steps). On instructor-led mode, whenever the performancemetric of the group exceeds a threshold, the instructor is notifiedabout the familiarity of students with the topic (e.g., making few or nomistakes on evaluation steps). If the conditions continue to hold(eventually, efficiency getting better) during a period of time, theonline learning system recommends shift of instruction method, and mayautomatically switch to a different learning mode.

In one embodiment, group activities may take into account additionalconstraints such as exploring preferred configurations and affinitylevels among students. Other learning modes may also be considered: forexample, best students may be selected for monitoring activities andpresenting seminars. Machine learning components may identify thresholdsused to indicate when learning modes should be executed. This can beperformed either automatically (fully based on learning data) or usingthe human in the loop (e.g., teacher provides a feedback on theperformance of the system).

For example, preferred arrangements and affinity levels may be assignedto positive scores; that is, whenever two students are put in the samegroup, a bonus value proportional to this score is assigned to the finalsolution. By exploring this aspect, an algorithm computing the groupsmay select arrangements with high scores. The selection of students formonitoring activities and seminars may follow decision-based ruleschemes. For instance, a student may only be selected if the studentreached the desired knowledge levels earlier than her colleagues. Anembodiment of a machine learning algorithm used to define thresholdvalues is as follows. For each possible threshold value, the learningsystem may collect the average rate of student improvement per timeperiod (e.g., hour, for example, average number of topics mastered perstudent and per hour) after the transition (and before the nexttransition); this information is extracted from historic data. Based onthe results, the learning system selects the threshold value deliveringthe highest expected improvement. If enough data is available, thisapproach can be extended to take into account different classes oftopics or student profiles.

FIGS. 3-5 illustrate examples of learning sessions performed by usersvia their devices in one embodiment of the present disclosure. Referringto FIG. 3, the figure shows four student devices 302, 304, 306, 308 andan instructor's device 3410. An online learning system of the presentdisclosure in one embodiment orchestrates an online learning session.For example, a user such as an instructor may log into the onlinelearning system as an instructor via that user's device, also referredto as an instructor's device 310. A learner or student may log into theonline learning system as student on a device (e.g., 302, 304, 306,308). The instructor via the instructor's device 310 may orchestrate alearning session. Consider that the online learning session selectsself-paced mode to start. The online learning system monitors each ofthe students' interactions with the activities presented on userinterfaces of the devices (302, 304, 306, 308) and determinesperformance metrics or measurements associated with each student inperform a given activity. The example in FIG. 3 shows five activitiesthat are monitored that are performed by each student. The monitoringindicates that the students as a group are overall doing well (onlystudent at 402 and 406 have not met an individual learning threshold asshown by ‘x’; all others have done well as shown by a check mark).Moreover, in the example shown, heterogeneity threshold is not exceeded.Therefore, the online learning system recommends and/or automaticallyswitches a mode of instruction to a self-paced mode. In one embodiment,two thresholds may be employed: (1) a learning threshold (which is samefor all students), for checking weather students are performingsatisfactorily; and (2) a heterogeneity threshold (which is same for allstudents), for checking whether one or more students are performingbetter than others.

Referring to FIG. 4, the figure shows four student devices 402, 404,406, 408 and an instructor's device 410. Consider that an onlinelearning session is conducted in self-paced mode, for example, followingthe example shown in FIG. 3. The online learning system continues tomonitor each of the students' interactions with the activities presentedon user interfaces of the devices (402, 404, 406. 408) and determinesperformance metrics or measurements associated with each student inperform a given activity. The example in FIG. 4 shows five activitiesthat are monitored. The monitoring indicates that the students as agroup overall are not doing well (each student at 402, 404, 406, 408)met an individual learning threshold or criteria with respect to onlyone activity as shown by one check mark; all other activities areillustrated with ‘x’ marks. Given that a high number of performancemetrics has not met the learning threshold, heterogeneity threshold isalso not met. Therefore, the online learning system here recommendsand/or automatically switches the mode of instruction to mediated, inwhich the student can learn together synchronously with the instructor,with the instructor guiding the activities on the instructor's device410 in sync.

Referring to FIG. 5, the figure shows four student devices 502, 504,506, 508 and an instructor's device 510. The online learning systemmonitor each of the students' interactions with the activities presentedon user interfaces of the devices (502, 504, 506. 508) and determinesperformance metrics or measurements associated with each student inperform a given activity. The example in FIG. 5 shows five activitiesthat are monitored. The monitoring indicates that the students at 502and 504 are not doing well with the five activities while the studentsat 506 and 508 are doing well. The online system determines that theperformances exceed the heterogeneity threshold. The online systemrecommends and/or automatically switches the mode of instruction to agroup activity mode. For example, group activity may be a preferred modewhen the heterogeneity is high: the students with better performance mayhelp the students with worse performance. The instructor via theinstructor device 510 may start group sessions. For example, groupsessions may be triggered whenever the heterogeneity level becomes toolarge (based on comparing to a threshold value that is predefined). Anexample criterion for grouping is coverage; for example, having groupscontaining students who, together, have sufficient knowledge level ofmost (if not all, for example 90% or another defined ratio orpercentage) topics. In this particular example, s1 and s4 may be put ina group (they know different things), but all students may be groupedtogether since as they together would know more. The number of groupscan be a pre-defined parameter of the problem, as well as the maximumnumber of students per group. The grouping problem may employ the setcovering problem, a classical combinatorial optimization problem forwhich several approximated and exact solution approaches are known.

FIG. 6 illustrates a schematic of an example computer or processingsystem that may implement a learning system in one embodiment of thepresent disclosure. The computer system is only one example of asuitable processing system and is not intended to suggest any limitationas to the scope of use or functionality of embodiments of themethodology described herein. The processing system shown may beoperational with numerous other general purpose or special purposecomputing system environments or configurations. Examples of well-knowncomputing systems, environments, and/or configurations that may besuitable for use with the processing system shown in FIG. 6 may include,but are not limited to, personal computer systems, server computersystems, thin clients, thick clients, handheld or laptop devices,multiprocessor systems, microprocessor-based systems, set top boxes,programmable consumer electronics, network PCs, minicomputer systems,mainframe computer systems, and distributed cloud computing environmentsthat include any of the above systems or devices, and the like.

The computer system may be described in the general context of computersystem executable instructions, such as program modules, being executedby a computer system. Generally, program modules may include routines,programs, objects, components, logic, data structures, and so on thatperform particular tasks or implement particular abstract data types.The computer system may be practiced in distributed cloud computingenvironments where tasks are performed by remote processing devices thatare linked through a communications network. In a distributed cloudcomputing environment, program modules may be located in both local andremote computer system storage media including memory storage devices.

The components of computer system may include, but are not limited to,one or more processors or processing units 12, a system memory 16, and abus 14 that couples various system components including system memory 16to processor 12. The processor 12 may include a module 30 that performsthe methods described herein. The module 30 may be programmed into theintegrated circuits of the processor 12, or loaded from memory 16,storage device 18, or network 24 or combinations thereof.

Bus 14 may represent one or more of any of several types of busstructures, including a memory bus or memory controller, a peripheralbus, an accelerated graphics port, and a processor or local bus usingany of a variety of bus architectures. By way of example, and notlimitation, such architectures include Industry Standard Architecture(ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA)bus, Video Electronics Standards Association (VESA) local bus, andPeripheral Component Interconnects (PCI) bus.

Computer system may include a variety of computer system readable media.Such media may be any available media that is accessible by computersystem, and it may include both volatile and non-volatile media,removable and non-removable media.

System memory 16 can include computer system readable media in the formof volatile memory, such as random access memory (RAM) and/or cachememory or others. Computer system may further include otherremovable/non-removable, volatile/non-volatile computer system storagemedia. By way of example only, storage system 18 can be provided forreading from and writing to a non-removable, non-volatile magnetic media(e.g., a “hard drive”). Although not shown, a magnetic disk drive forreading from and writing to a removable, non-volatile magnetic disk(e.g., a “floppy disk”), and an optical disk drive for reading from orwriting to a removable, non-volatile optical disk such as a CD-ROM,DVD-ROM or other optical media can be provided. In such instances, eachcan be connected to bus 14 by one or more data media interfaces.

Computer system may also communicate with one or more external devices26 such as a keyboard, a pointing device, a display 28, etc.; one ormore devices that enable a user to interact with computer system; and/orany devices (e.g., network card, modem, etc.) that enable computersystem to communicate with one or more other computing devices. Suchcommunication can occur via Input/Output (I/O) interfaces 20.

Still yet, computer system can communicate with one or more networks 24such as a local area network (LAN), a general wide area network (WAN),and/or a public network (e.g., the Internet) via network adapter 22. Asdepicted, network adapter 22 communicates with the other components ofcomputer system via bus 14. It should be understood that although notshown, other hardware and/or software components could be used inconjunction with computer system. Examples include, but are not limitedto: microcode, device drivers, redundant processing units, external diskdrive arrays, RAID systems, tape drives, and data archival storagesystems, etc.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements, if any, in the claims below areintended to include any structure, material, or act for performing thefunction in combination with other claimed elements as specificallyclaimed. The description of the present invention has been presented forpurposes of illustration and description, but is not intended to beexhaustive or limited to the invention in the form disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the invention.The embodiment was chosen and described in order to best explain theprinciples of the invention and the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated.

We claim:
 1. A method of online learning, the method executed by atleast one hardware processor, the method comprising: selecting contentfor a learning session and opening a user interface to start thelearning session on a plurality of devices; selecting a mode ofinstruction for the learning session, wherein the mode of instructioncomprises one of self-paced learning and mediated learning; presentingvia the user interface an activity to perform associated with thecontent during the learning session; retrieving from a storage device akey performance indicator associated with the activity, a target rangeassociated with the key performance indicator, and heterogeneitythreshold associated with the activity; monitoring performance of theactivity, and generating a performance metric associated with the keyperformance indicator for the activity performed on each of theplurality of devices based on the monitoring; and responsive todetermining that the performance metric is outside of the target range,automatically switching the mode of instruction to a different mode ofinstruction, and presenting on the user interface, the activity to beperformed in the different mode of instruction.
 2. The method of claim1, wherein the activity comprises one or more of definition of terms,text material, multi-media content, quiz, questionnaire and test.
 3. Themethod of claim 1, wherein if the mode of instructions comprises theself-paced learning, the different mode of instruction comprises one ofthe mediated learning, group activity learning, and pair activitylearning.
 4. The method of claim 1, wherein if the mode of instructioncomprises the mediated learning, the different mode of instructioncomprises one of the self-paced learning, group activity learning, andpair activity learning.
 5. The method of claim 1, wherein responsive todetermining that the performance metric is outside of the target range,sending a notification signal to a device of a user that is leading theonline learning session.
 6. The method of claim 1, wherein the keyperformance indicator comprises one or more of time to perform theactivity, error rate in performing the activity, accuracy rate inperforming the activity.
 7. The method of claim 1, wherein themonitoring comprises determining an amount of time the activity ispresented on the user interface until a next activity is presented onthe user interface.